Skip to content

fix(evidence): stop calling a limitation "fitted with" when it says "not fitted" - #1173

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:fix-evidence-limitation-prefix-046
Oct 14, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:fix-evidence-limitation-prefix-046

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

laya/evidence.py::inspect_calibration prefixed every recorded calibration limitation with
"fitted with recorded limitations: ", then quoted the limitation text straight after it. One of
the texts it quotes reads "not fitted: ...", so a single line of laya-evals evidence output
asserted the fit and denied it.

  • laya/evidence.py +4/-1 (@@ -60,7 +60,10 @@) -- the prefix is now "limitations recorded: ",
    plus a three-line comment on why the prefix claims nothing.
  • tests/test_evidence.py +17 (@@ -52,6 +52,23 @@) -- four checks on the reason text, which
    nothing in that file used to assert at all.

No state, exit code, detail payload or function signature changes. A type that was INSUFFICIENT
is INSUFFICIENT before and after, with the same items and issues in detail.

Why

The artifact being read is written by laya/train.py::calibration_report, whose first issue bullet
is "fewer than MIN_TYPE_N items: the temperature was not fitted and stays 1.0", emitted as
"not fitted: %d calibration items, fewer than %d, so the temperature stays 1.0". evidence.py
wraps that string in a sentence that says the opposite. Two processes, the same config, output pasted
exactly as evidence.format_summary printed it -- first from a tree extracted from c73908d9
(release 0.4.6), then from this branch (the process asserted on its own laya/evidence.py path before
printing, so each block is that tree's output):

evidence summary for synthetic
(reports evidence and limitations found in existing artifacts; this is not a deployment verdict)
calibration: INSUFFICIENT
  choice                   INSUFFICIENT -- fitted with recorded limitations: not fitted: 5 calibration items, fewer than 10, so the temperature stays 1.0
  score                    INSUFFICIENT -- fitted with recorded limitations: fitted on only 30 calibration items
  noul                     PRESENT -- fitted with no recorded issues on 400 items
evidence summary for synthetic
(reports evidence and limitations found in existing artifacts; this is not a deployment verdict)
calibration: INSUFFICIENT
  choice                   INSUFFICIENT -- limitations recorded: not fitted: 5 calibration items, fewer than 10, so the temperature stays 1.0
  score                    INSUFFICIENT -- limitations recorded: fitted on only 30 calibration items
  noul                     PRESENT -- fitted with no recorded issues on 400 items

Every state is identical, the noul line is byte-identical, and so is the overall line. The choice
line is the bug: INSUFFICIENT is right, and the words before the quoted limitation asserted the
opposite of the words after it.

That matters more here than a wording nit usually would, because this module's own docstring states
the rule it breaks: "Every state it assigns is either derived directly from an upstream artifact or
marked UNKNOWN. It never manufactures a healthy-looking state from absent input."
The prefix is the
one piece of the reason line that is not from the artifact, and for the sub-MIN_TYPE_N type it is
a claim the artifact explicitly refuses to make.

The prefix is safe to reword: grep -rn "fitted with recorded limitations" over laya/, tests/
and docs/ on c73908d9 matches only laya/evidence.py:63 -- nothing parses it, no test asserted
any reason text (grep -c reason tests/test_evidence.py is 0 on main), and no doc page shows it.
docs/evals.md documents the command without quoting its reason lines.

How it was verified

python tests/test_evidence.py
python scripts/test_suites.py
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya/ tests/
zensical build --strict --clean

tests/test_evidence.py goes 40 (all 40 checks passed, from a tree extracted from c73908d9,
laya/evidence.py and tests/test_evidence.py byte-identical across 0.4.4/0.4.5/0.4.6 --
f88d4b8e and 887e6a65) -> all 44 checks passed. Full gate all 87 suites passed, rc 0, its pytest leg
500 passed, 1 skipped in 15.73s, and 500 passed, 1 skipped on the extracted tree as well
(16.69s) -- unchanged, as expected: tests/test_evidence.py is a script suite in
scripts/test_suites.py and the 500 counts the separate PYTEST_SUITES invocation, so the number this
change moves is that file's own 40 -> 44. ruff rc 0, compileall rc 0, docs strict build No issues found rc 0
in 1.44s. The evidence summary appears in no built page
(grep -rn "inspect_calibration\|limitations recorded" docs/ is empty on this branch), so the strict
build is a regression check here rather than a doc update.

The four new checks are the three limitation kinds' full reason strings asserted verbatim --
below-MIN_TYPE_N, below-CALIB_WARN_N and clamped, named
<kind> reason is the prefix plus the upstream text, verbatim -- plus one rule check,
no limitation reason states a fit the artifact did not record, that no limitation reason starts with
"fitted". Verbatim rather than substring, because the bug was in the prefix and an
assertIn(issue_text) would have passed with any prefix.

Mutation check, tests/test_evidence.py after each edit, laya/evidence.py restored with
git checkout -- between runs (branch committed first, git status clean before each run):

mutant result
control: no edit all 44 checks passed
old prefix "fitted with recorded limitations: " restored 4 kills -- the three verbatim checks plus no limitation reason states a fit the artifact did not record, which is the only one of the four that reads the shape rather than the exact string
"recorded limitations: " (harmless reword, no false claim) 3 kills -- the verbatim checks only; the rule check survives, which is the point of having both
reason reduced to the constant "limitations recorded upstream", dropping the quoted upstream text 3 kills -- the verbatim checks, each reporting got 'limitations recorded upstream', expected 'limitations recorded: <upstream text>'

So the verbatim pins are what notice a silent reword, and the rule check is what would notice the
specific class of reword that re-asserts a fit.

Deconfliction: 49 open PRs as at 2026-10-12T13:12:34Z, and the whole open set's file lists pulled
and matched line by line -- none of them edits laya/evidence.py or tests/test_evidence.py.

Not fixed here, worth knowing

  • The PRESENT branch still reads "fitted with no recorded issues on %d items". That one is
    defensible and deliberately left alone: for an artifact calibration_report actually wrote, an
    empty issues list with items > 0 can only mean at least MIN_TYPE_N items, because the
    sub-floor case always records its own issue. So "fitted" follows from the writer's invariant
    there, where in the INSUFFICIENT case it contradicted the writer's own sentence. A hand-written
    config with {"items": 5, "issues": []} does get that claim, and the loop at c73908d9
    tests/test_evidence.py:72-75 (lines 89-92 on this branch, below this change's insertion)
    deliberately pins the state for that shape without judging the count -- keeping the module at one
    numeric source of truth, as feat(evals): evidence inspection over checkpoint and eval artifacts #964 intended.
  • {"laya_train_calibration": {}} yields overall INSUFFICIENT with no reason at all, while a
    missing calibration block yields UNKNOWN with a reason. That asymmetry is a state question,
    not a wording one, and belongs to whoever owns the ladder.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main (c73908d9, release 0.4.6)
  • Tests pass locally
  • Docs or examples updated when the public API changed

…not fitted"

`inspect_calibration` prefixed every recorded limitation with
"fitted with recorded limitations: ", then quoted the issue text from
`laya.train.calibration_report` -- whose sub-`MIN_TYPE_N` entry reads
"not fitted: 5 calibration items, fewer than 10, so the temperature stays 1.0".
One `laya-evals evidence` line therefore asserted the fit and denied it in the
same breath, which is the one thing the module docstring promises it never does
(assign only states derived from the upstream artifact).

The prefix now says "limitations recorded: " and adds no claim of its own.
`tests/test_evidence.py` pinned calibration states only -- it had no check on
`reason` at all -- so the three limitation kinds' full reason strings are now
asserted verbatim, along with the rule that a reason never starts with "fitted".
@NandhaKishorM
NandhaKishorM merged commit 9564bdf into NandhaKishorM:main Oct 14, 2026
44 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged in 0.4.7.

The line asserted a fit and denied it in the same breath: calibration_report writes not fitted: 5 calibration items, fewer than 10, so the temperature stays 1.0 into the very list the prefix was calling "fitted with". A reader of laya-evals evidence had no way to tell which half to believe.

"limitations recorded: " is the right shape because the reason's job here is to carry the upstream text, not to add a claim of its own. The three verbatim checks plus not t["reason"].startswith("fitted") pin that in both directions, and the PRESENT branch keeps saying "fitted with no recorded issues" where a fit really did run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants